With the advent of broadband, there is an increasingly large volume of video, audio, and other multimedia content being posted to the Internet and the World Wide Web (“web”). A significant portion of multimedia content is delivered via dynamic formats to web browsers with the appropriate dynamic format modular interface (“plug-in”). Unlike regular web pages, dynamic formats may not be programmed in Hypertext Markup Language (“HTML”). Rather, the code is typically compiled into a different format, which is embedded on a web page and played by a web browser's appropriate plug-in. Certain other multimedia products also follow a similar delivery pattern.
Web crawlers are programs that locate web content. While dynamic formats provide a rich, interactive experience, they present a challenge for web crawlers. Traditional web crawlers are built to parse HTML and other text content by following hypertext links, and automatically submitting forms in order to crawl database-driven information that is not accessible by following links alone. But traditional web crawlers are not able to fully crawl all relevant data upon encountering many web pages in dynamic formats.